Some robots may be controlled using relatively abstract commands such as “pick up object at location A and carry to location B.” This level of abstraction may be well-suited for specialized robots operating in relatively predictable environments such as factories. However, it may be less well-suited for controlling general-purpose robots in less predictable environments such as disaster zones, public spaces, outdoor environments, or even homes/offices. For example, a child's room may be cluttered with multiple heterogeneous toys that need to be picked up and put away in order for the room to be deemed “clean.” Dirty dishes may need to be carried from the dinner table to the sink and/or loaded into a dishwasher. Requiring an operator to individually instruct the robot to pick up each dish and take it to the sink would be inefficient and frustrating for the operator.